Smart hub

ABSTRACT

An intelligent hub for interfacing with other devices and performing smart audio or video source selection.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/400,142, which was filed on Jan. 6, 2017; U.S. patentapplication Ser. No. 14/597,863, which was filed on Jan. 15, 2015, byJames A. Proctor, Jr. for a SMART HUB and claims the benefit of U.S.Provisional Patent Application Ser. No. 61/928,583, which was filed onJan. 17, 2014, by James A. Proctor, Jr. for a SMART HUB and all of whichare hereby incorporated by reference.

This application is related to the following issued U.S. Pat. Nos.8,385,913; 8,090,616; 7,936,736; 8,090,359; 8,116,749; 8,370,955;8,385,896; 8,374,592; and 8,369,842 and is also related to the followingU.S. patent application Ser. No. 13/744,457 filed Jan. 18, 2013; Ser.No. 13/775,435 filed Feb. 25, 2013; Ser. No. 13/449,208 filed Apr. 17,2012 and U.S. Provisional Application Ser. No. 60/874,992 filed Dec. 15,2006, entitled “Wireless event indication system and method”. The entirecontents of the above-referenced patents and patent applications arehereby incorporated by reference.

SUMMARY

This application relates to a smart hub.

BRIEF DESCRIPTION OF THE DRAWINGS

The details will be apparent from the following more particulardescription of example embodiments, as illustrated in the accompanyingdrawings in which like reference characters refer to the same partsthrough the different views. The drawings are not necessarily to scale,emphasis instead being placed upon illustrating embodiments.

FIG. 1 is an example system diagram of a preferred embodiment.

FIG. 2 is a block diagram depicting an example architecture of a smarthub (SH) embodiment.

FIG. 3 is a block diagram depicting an example architecture of anembodiment of a smart hub remote control (on a smart hub applicationdevice, referred to as the SHAD embodiment).

FIG. 4 is an example software/firmware architecture functionalpartitioning diagram of the processing performed in an embodiment of asmart hub.

FIG. 5 is an example software/firmware architecture functionalpartitioning diagram of the processing performed in an embodiment of amobile device smart hub remote control/interface application (residenton an embodiment of a smart hub application device—SHAD).

FIG. 6 is an example software/firmware architecture functionalpartitioning diagram of the processing performed in an embodiment of amobile device smart hub server application.

FIG. 7A is an example flow diagram of a smart hub application device(SHAD) embodiment detecting the presence of a smart hub (SH) or arelated devices (such as a smart but universal remote extender, or theanother device) and invoking an embodiment of a smart hub interfaceapplication.

FIG. 7B is an example system level flow diagram of a smart hub systemembodiment wherein a user with a smart hub application device (SHAD)embodiment comes into range of a smart hub embodiment to perform exampleservices.

FIG. 8 is an example system level flow diagram of a smart hub embodimentperforming a embodiment of a source switching process.

FIG. 9A is an example illustration of utilizing combined Acoustic and RFProcessing to determine information related to the a SHAD device.

FIG. 9B is an example system flow diagram a smart hub system processembodiment utilizing combined Acoustic and RF Processing for thedetermination of Acoustic channel state information (CSI) (including oneor more of: presence in a specific area presence, location (proximal orabsolute), acoustic channel equalization, spatial equalization, and thelike).

FIG. 10 is an example block diagram Smart Hub Universal Remote Extender(SHURE) embodiment (allowing for a smart hub to provide IR control (orother control) to controllable devices providing content sources to theswitchable ports of a smart hub, when such devices are not directlycontrollable by resources internal to the smart hub components), in oneembodiment.

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example system diagram of a preferred embodiment.

Referring to FIG. 1, an example embodiment of a smart hub system isdepicted. The Smart hub 110, in the current embodiment, is configured toperform a number of functions and communicate with a variety of otherdevices and network entities. For example the smart hub 110, in thecurrent embodiment, is in communication with a local computer or localserver 103 utilizing application communication link 117. It should beunderstood that application communication link 117 may be performedutilizing any combination of wired and/or wireless physicalcommunication links such as utilizing wired Institute of Electrical andElectronic Engineers (IEEE) 803.2 Local Area Network (LAN) connections,802.11 Wireless Fidelity (Wi-Fi) connections, Bluetooth® connections,and the link. The individual physical communication links may further bedirectly between the smart hub 127 and the local server 103 (such aswould be the case with so-called Wi-Fi Direct®), or may utilizeintermediary network nodes such as router 105.

Likewise, application communication link 116 between the smart hub 110and a smart hub cloud server at cloud 102 may be enabled by any numberof individual communications paths, such as utilizing cellular mobilecommunications via base station 101, wireless and wired router 105,Wi-Fi direct, Bluetooth, Near Field Communications (NFC), Ultra Wideband(UWB) or the like. While wired LAN connections to the router 105 are notdepicted in FIG. 1, such connections are contemplated and may be used invarious embodiments. In one embodiment, communications between the smarthub cloud server (SHCS) and the smart hub (SH) via applicationcommunication link 116 are performed utilizing a smart hub applicationdevice (SHAD) as an intermediary and utilizing individual communicationscapabilities in between the SHAD 114 and the SHCS, as well utilizingindividual communications capabilities in between the SHAD 114 and theSH 110. In embodiments, application link 116 may be tunneled throughapplication links 118 and 115, or application link 116 may be simplyrouted or forwarded utilizing the physical communication capabilities ofthe SHAD 114, or another intermediate device such as router 105. In oneembodiment the SHAD utilizes a local wireless link 127 utilizingBluetooth high-speed (BT-HS) or Bluetooth LE (low energy), standardBluetooth in one embodiment, or 802.11 Wi-Fi between the SH and a SHAD114 (in one embodiment being a smart phone which is running a smart hubapplication) and forwards or otherwise enables messaging in-between theSHCS and the SH. The SHAD 114 additionally performs communications withthe SH utilizing application link 118 for enabling various functionalityto be described below, which requires intercommunication between thesetwo devices. It is contemplated that the means for such applicationlevel communications (115, 116, 117, 118, or other applications link notexplicitly shown) would in many cases utilize Bluetooth or Wi-Fi, or theassociated variants together or apart. Other communication technologiesare contemplated as well for use in the forgoing links, includingcellular, 4G, LTE, 3G, WCDMA, HSPA+, 1×EV-DO, CDMA2000, NFC, UWB,acoustics, ultrasonics, infrared, irDA, and the like. Similar physicalcommunications technologies are contemplated for enabling theapplication link 115 between the SHCS and the SHA as well, including theuse of 4G LTE, or 802.11 Wi-Fi via router 105.

In one embodiment, router 105 is a so-called femto cell and provides 4Gor other cellular technologies (WCDMA, 1×EVDO, etc.) as a localinterface to the SHAD 114, or SH 110, or another device, with aninternet connection (via links 130 and 129) to the cloud 102. Cloud 102in some embodiments is a public internet, while in other embodiments maybe a private network such as those operated within a large serviceprovider such as AT&T, Time Warner, Comcast, or the like.

The mobile device, having a smart hub application, is one embodiment ofa smart hub application device (SHAD) 114. It is contemplated thatvarious embodiments of a SHAD may operate within the embodiments ofFIG. 1. One alternative SHAD is a standalone and purpose built smartremote device, explained in more details associated with embodimentsdescribed associated with FIG. 3, FIG. 5, FIGS. 7A and 7B, FIGS. 9A and9B and elsewhere. In one embodiment, SHAD 114 utilizes Bluetooth LEwireless capability and is capable of detecting Bluetooth beacons (orWi-Fi) such as those contemplated in related application.

Smart Hub Source Selection and Switching

An embodiment of the smart hub (SH) 110 operates as a smart switch,allowing for a number of services to be discussed, including in someembodiments acting as a universal remote control, together with (orindividually) acting as a smart and predictive content source switch(High Definition Multimedia Interface (HDMI) ports for example). In oneembodiment the SHAD 114 provides for a user interface to provide formanual control of the SH universal remote control functionality, as wellas the HDMI port selection to be output to HDMI input of televisionmonitor 111. The SH may additionally be capable of selecting othersource inputs such for output to the monitor 111, such as streamingaudio (Pandora® for example), streaming video (Netflix®, Hulu®, Amazon®Prime for example), remote display content (Apple's Airplay®, UPnP, orGoogle's Chromecast® for example), content from one or more USB ports(mp3, mp4, jpg files), or a local or remote server (iTunes®, iCloud®,Dropbox®, Skydrive®, Google® Drive, Amazon®, a shared NTFS or directory,or the like), or other sources.

Smart Hub Universal Remote Control Feature

In some embodiments, the SH includes a universal remote controlfunctionality and is capable of controlling the power on/off, channel,volume, pause/play, rewind, fast forward, channel switch, picture inpicture, DVR recorded content selection, one-demand content selection,track selection, chapter selection, and other traditional remotecontrollable features of a device providing content to one or more portsof the Smart hub 110. The universal remote control functionality of theSH 110 in some embodiments extends to the control of the other forgoingcontent sources, such as streaming or local sources. Such control mayinclude performing a series of steps in combination to achieve a desireduser selection or autonomously determined source selection and enablingof the source content. For example, a login username and password may beprovided to a streaming service, or local network credentials providedto a local computer for gain access to iTunes® content or the like, andcommands exchanged with content servers to effect desired actions(pause, fast forward, rewind, program, movie, song selection, etc.).Combinations of so called “soft” universal remote control capabilitiesapplying to the delivery of streaming or other non-local contentdevices, and/or a number of hard remote control steps (to local devicesproviding HDMI or other input to a physical port or the SH) may becombined in series to achieve specific goals associated with userdesired or associated with determined content switching events. As oneexample, the SH determines a switch should occur. Such a determinationmay be performed predictively, or with direct command from SHAD 114, orpredictively and with confirmation from the SHAD 114 user inputauthorizing a suggested switch. Associated with a switching event, theSH, in one embodiment, would power off the previously used deviceproviding input to a HDMI input for example, and power on and sendspecific other related commands (setting the display to recordedprograms listings on the digital video recorder (DVR)/Cable box forexample) to a second source device, then perform the switch to the newHDMI port, in a coordinated manner. In one embodiment, the new sourcemay be a streaming source such as Netflix®, and there may be delay ingaining access to the new content, and the switch from and power off ofthe previous content source may be delayed until the new content isavailable as a stream, or at an input port of the SH 110 device (in thecase of a HDMI interface from a DVR/cable box for example). In someembodiments, specific universal controller commands may also controloutputs of the content, such as the monitor 111 (input HDMI port to themonitor 111, volume, audio output source, audio input source, source,picture settings, display settings, and the like), and Audio system 112settings (volume, input port, processing settings, equalizer settings,output settings, etc).

The configuration of such controllable devices may be set using severalmethods, for example the SH 110 may perform an search for controlprotocols and examine the HDMI or other input ports for activityassociated with a specifically transmitted remote command so as toautomatically determine the specific control protocols and with feedbackbased upon monitored device inputs to the SH 110 from Sources 1 to n.The discovery and configuration of a Smart Hub Remote Extender (SHURE)120, or additional SHUREs may be performed in similar approaches andincluding interaction with a user of SHAD 114 to perform confirmationsteps or provide required information to allow for configuration. Thedetection of a specific SHURE by SH 110 may be performed in an approachsimilar to the detection of a SH 110 by a SHAD 114 using transmittedbeacons for example from one or more of the devices (iBeacons® in oneembodiment, and/or beacons as described in the related application(s)referenced above.

Learned Behavior by the Smart Hub 110

The Smart Hub 110 may qualify and respond to “events”. Events can beactivity after long pauses, learned behavior of a user such as regularlywatched programs, etc. After long pauses in activity of the Smart Hub,if a Remote device is detected (event), autonomously switch to the userpreferred source. The Smart Hub 110 can look for patterns such asregularly watched shows and autonomously switch to regularly watchedprograms at the time of day when the program is aired based on userpreference and what users/user devices are detected in the room.

Fuzzy switching concept. Beacons. The Smart Hub 110 emits a Bluetoothbeacon. When a smart phone or second device configured to communicatewith the Smart Hub 110 enters the local area of the Smart Hub 110, thebeacon causes an inactive application within the smart phone/2nd deviceto launch and engage with the Smart Hub 110. The smart phones responseto the beacon may influence switching or switching preferences.Providing program specific content such as live commentary about thecurrently running program, use of twitter or Facebook, or other socialsite with the second screen app, take polls, push advertisements oroffers, or program information. Beacons may also be used by the SmartHub to learn whose devices are located within the local area and adjustpreferred settings based on which users are located in the area. TheSmart Hub 110 can be configured to switch sources through manual inputvia a physical button on the remote, smartphone, or other device. TheSmart Hub 110 may be controlled by a text message, twitter message,Bluetooth signal, WiFi, or Ethernet carried message. The Smart Hub maybe controlled via the microphone located on either the Smart Hub,Remote, smart phone, or smart device by voice recognition or even use ofa “clapper”. The Smart Hub received SMS messages from a plurality ofaddresses that can be prioritized, allowing for protection fromoverride, or to allow for an override of other inputs from otheraddresses. A name associated with a port (HDMI port for example) and aname associated with the SMS address (“Hubbie” for example). Ex: “SIRI,new message to Hubbie, switch to HDMI 1”.

Device Discovery

As described associated with FIG. 7A, the SHAD 114 discovers thepresence of SH 110 by detecting the transmission of beacons from the SH110. (In alternative embodiments, the SHAD 114 may transmit beacons tobe detected by SH 110, and/or SHURE 120.) The SHAD 114 detection ofbeacons (in one embodiment by the operating system of the device of aspecific identifier in a Bluetooth transmission called a proximity UUIDor universally unique identifier, and other information such as a majorand minor value as defined for iBeacons) provides for the invocation ofthe smart hub application on the SHAD 114, and allows for theperformance of a number of actions. Some of the actions involve thedetermination and display of specific user input information, which maybe context dependent. For example one display of the device may be assimple as notifying the user they are now in range of the SH 110, and ifthey would like to display the smart hub application interface for use.Other interfaces screens may be dependent upon the specific informationdetermined as a result of communication with the SH 110 using link 118,and may include a request for the confirmation of a potentially andsuggested desired content switch. One such example would be, the startof a live sporting event or other program, and the suggesting oswitching from another HDMI input (XBOX for example) to the cable boxoutput to the input of the SH 110 as to display the live contents on thedisplay 111. Such indication may be simultaneously displayed orexclusively displayed on the monitor 111 as well. Other screens aredisclosed in forthcoming descriptions, and include in one embodiment:

-   -   Universal remote control interface    -   switcher control    -   second screen services    -   Social Interface    -   System configuration service    -   Sound interface    -   Services interface    -   Advertising interface    -   Offers interface    -   Incentives interface        Content Identification

The SH application in the SHAD 114 may also be capable in specificembodiments of presenting so called “second screen” information to theuser, and the information may be dependent upon the specific contentselected by the smart hub 110. The smart hub may determine the identityof the content using audio or video based fingerprinting techniques asknown in the art, and using SHCS 103 or the cloud servers accessible bylink 116, or by internet cloud 102, via cellular system 101, localnetwork router 130, and WAN interface 104, or other means as described(via interface to SHAD 114 using link 118 for example).

The SHCS 103, in some embodiments, may perform the identification of theselected, or other available source content so as to determine andassociated information relating to both currently playing, or availablesource content (for example on another HDMI input port to the SH 110, oron a DVR, of streaming service screen). Examples of second screeninformation available to the SHAD 114 application to present to the userinclude advertisements, offers, loyalty points, social content (liveTwitter® hashtag content associated with the specific SH displayedcontent, or other available content, ratings of the content fromInternet Movie Database (IMDB.com) or like sources—which can be used asinput to suggesting a switching proposal to user in some embodiments,offers for the purchase of related content—music from the show, apparelform the show, a DVD or movie download or digital rights, an offer tobarter for other content as described in the related application(s),incorporated herein by reference).

In one example, games may be played amongst users of defined groupswatching the same content (for example all users watching he content,only friends belonging to a friend list, only people in the same room,or location—using a single SH, or set of SHs.) Such games might includepolls associated with guessing potential outcomes of the plot of a showor movie, then displaying who got it right, and wrong, or what all theusers collectively think will happens, or thought would happen, andcomparing the predictions with what actually did occur.

One further example is predicting details of a live sporting event, andshowing a graphical representation of the collective group thought, asit currently stands in favor of one team or the other. Other examplesinclude polls associated with one event among a plurality ofevents—presidential election predictions and/or results for example. TheSHCS in some embodiments may further collect statistics associated withviewing behavior of users who are identified as being present in theroom of the smart hub 110 while specific content is being played, or ifnot individually identified, collecting generally content statistics.Such information being usable for targeted advertisements or offers, orother uses where knowledge about who is watching currently, or generallywould be usable for more effective communications and offers.

A server may back up the settings from the Smart Hub and is logicallyconnected to the Smart Hub 110 in one embodiment. The server may receiveaudio or video samples and identify the content being watched. Theserver may receive “samples” as reconstructable coded information orencoding or hashing the content in a non-reconstructable way so as toallow for identification, but not necessarily recreation. Performing theidentification of the shows or audio content my take place locally onthe Smart Hub based upon downloaded “signatures” to hashed contentinformation or the identification may be performed on the server.Identifying the content may be based upon information encoded within thecontent stream itself, such as metadata, or information packets, ordigital channel guide information, or interacting with the contentproviding Smart Hub itself (Ethernet, or HDMI interaction).

Smart Hub Universal Remote Extender (SHURE)

In some embodiments, the local devices are controllable by infrared andknown protocols and standards (IrDA for example), but are “blocked” byphysical obstruction so as to prevent the remote control signal 127, 131(as an IR signal in one embodiment) from the SH 110 being able tocontrol a specific device (for example source 2 108 being a cable boxincluding a DVR in one embodiment). In such as case, a physical wirewith an IR transceiver or simple emitter may be employed as known in theart. In one embodiment the SH 110 may send Bluetooth commands (a oneembodiment of link 124) to a smart hub universal remote control extender(SHURE) 120, which provides IrDA commands (as one embodiment of link125) to the associated “blocked” device/source 2 in the forgoingexample, each step being performed wirelessly, and in some embodimentsthe SHURE 120 is battery powered.

A block diagram of an embodiment of the SHURE 120 is shown and furtherdescribed in connection with FIG. 10. Alternatively, in someembodiments, SHURE 120 may receive commands to control the sources 1 ton (107, 108 . . . 109) and/or Audio system 112 directly from SHAD 114.

User Input and Confirmations on SHAD, and Example Uses

The input to the SHAD 114 for the confirmation of a switching event, orto invoke another actions such as invoking a switching event to thenext, or a specific input or source may be based upon use of a motionsensor, proximity sensor compass, or accelerometer in the SHAD 114 so asto allow for tapping twice for confirmation, or no input/a differentinput for denial of a suggested switch. As a result confirmation may beperformed without taking the device out of a user's pocket. In onescenario, the SH 110 determines a switch of input source may be desired,and displays a notification on the screen for a period of time, and willonly perform the suggested switch of one or more users having SHADdevices tap, shake or move their SHAD 114 as confirmation (in variousembodiments). The various SHAD sensors, as mentioned above and typicalin smartphones such as the iPhone® 5s may further be usable to enablegames to be played individually or as a group. The SH 110, using theuniversal remote or other capability may translate commands into theappropriate interface for a Wii®, XBOX®, or other gaming content sourceor platform, further the SH 110 may provide for games to be playeddirectly on the processor of the SH, or remotely on the SHCS 103, or thelike. In some embodiments, the SH may display simple games (Packman®,Pong®, etc.) directly over the content on the screen, for example duringcommercials. Such commercial periods may be determined by userconfirmation as a result of suggestions by the SH in one embodiment. Inother embodiments, the SH may perform commercial period detectionautonomously, and perform a switch to other content with no input from auser, or with confirmation as discussed or in another conformationapproach. In one example the commercial is displayed in a picture inpicture mode by the SH, and another gaming platform is displayed for theusers to play during the commercial (using traditional platform gamecontrollers, or alternatively using the SHAD devices as discussed).

Returning to the main program may again be performed via suggestion bythe SH and confirmation on the SHAD, or simply by a specific input onthe SHAD 114. One such input might be waving the device side to side, orup and down, or in circles in a specific direction to provide specificinputs, tapping, or number of taps on the device to indicate variousdesired actions. For example tapping the SHAD on the table twice (or theback of the device twice with your finger) could indicate go back to theprevious “main show (and for example, two taps to confirm switching to asecondary source, and/or 4 or 5 taps to invoke switching back). In oneembodiment, the secondary content may be paused using the SH 110internal universal remote or universal remote extender 120, when themain show, or content is being displayed, such pausing occurringautomatically when the SH 110 returns to the “primary” or main showbeing watched. Such motion or touch based inputs and gestures may act asa general approach for navigating menus displayed on the monitor 111 bythe SH 110, again with the advantage of not having to remove the devicefrom a user's pocket to provide input (in the form of tapping in a knownpattern for menu down, up, select, back, for example).

User Identification, Presence and Location, and Acoustic Equalization

The presence and identification of a specific user (presumablyassociated with a specific SHAD, such as 114) may be performed byBluetooth LE “ranging” as available in IOS® 7, or other variants. A moreaccurate approach may be achieved using the SHAD microphone to time theacoustic transmission from one or more speakers 113-1 to 113-4individually or in combination.

Such a process is further described associated with FIGS. 9A and 9B.Range may be determined form each of the speakers by performing a timingreference point between the SH 110, and the SHAD 114 using wirelesstransmission(s) between them. As the wireless transmissions willpropagate significantly faster than any acoustic transmissions formspeakers 113-1 to 113-4, the vast majority of the delay will be a resultof the acoustic propagation from the speakers to the SHAD 114, to bedetected by the microphone inputs of the device (other than anyprocessing delays in the overall process which may be calibrated out).As a result the physical distance of the SHAD 114 from each of thespeaker may be accurately estimated, and the user (associated SHAD)distances from each speaker determined, given the know speed of sound,and accurate delay calibration of the system. If the geometry of thespeaker arrangement is known, the actual location of the user may bedetermined is a sufficient number of speakers are utilized (3 speakersfor actual location, and 2 speakers for two locations with an ambiguityof which location—which a ambiguity being resolvable with knowledge ofthe room potentially, or in combination with Bluetooth rangingapproaches.) If the geometry of the speaker arrangement is not known,the presence of the user (SHAD 114) in the room, versus the speaker notbeing in the room may be surmised based on distance, and the knowledgethat sound does not propagate through walls of the structure, as theBluetooth beacons will). The Bluetooth ranging in combination withacoustic ranging is usable to provide an estimation of user/SHADlocation as well, given the knowledge of the speaker geometry with theSH location. In an alternative embodiment, one or more USE 120 devicesmay be placed in known locations within the room, and used to performmultiple ranging processes between the SHAD 114, the SHURE 120, and theSH 110. Potentially additional techniques may be used to refine thepossible location of SHAD 114.

One approach to the calibration of the acoustic ranging approach is toinclude a microphone within SH 110 or one or more SHURE 120 devices,capable of receiving the acoustic transmissions from the audio system112 and associated speaker 113-1 to 113-4 (or any number of speakers inembodiments). When the SHURE 120 includes a microphone, the time delayfrom the transmission (or reception) or a BT or other wirelesstransmission to the reception of the acoustic signal may be determined.In one embodiment, the SHURE 120 having a mic is placed at the speakerlocation to determine an absolute timing reference of the delay totransmit the acoustic ranging signal. When the mic is in the SH 110, thepropagation from the speakers to the SH and the ambiguity are summed,known geometries may be used to calculate the actual calibration delay.In general specific embodiments, no calibration would be required andthe processing delay and wireless communications delay would benegligible in comparison to the acoustic propagation delay. The acousticsignal may be useable to so-called “channel measurement” or channelsounding to determine the signal processing “channel propagation state”or channel state information (CSI) between each speaker and the relevantSHAD 114 microphone (217 of FIG. 2). Such CSI may be frequency dependentand useable to perform acoustic equalization of the room so as tooptimize the audio experience for the user of SHAD 114. The acousticsignal may be sent standalone in with no other sound present or it maybe embedded in existing source media and performed real time andperiodically or continuously. Such a process ideally uses an acousticsignal which is not detectable by a user listening to the contentselected by SH 110. One approach to such a signal would be to embed alower level signal within the media acoustic signal in such as was as tominimize it detectability by a person listening, and the signal having awide bandwidth allowing for channel estimation. One such signal is anOrthogonal Frequency Division Multplex (OFDM) pilot as used in IEEE802.11 wireless systems for channel estimation, or a Zadoff-Chu or otherCAZAC code as used in LTE, or even Walsh codes as used in CDMA cellularsystems, or barker codes or CCK codes as used in IEEE 802.11. Suchsignals may be correlated with the know reference versions of the signalto determine the channel impulse or channel frequency response of theacoustic channel. A number of the lower level reference signals may beintegrated (or the results of multiple correlations would be integrated)to provide to processing gain and detectability, resulting in a signalto noise ration allowing for quality channel estimation and presencedetermination. Another approach, in an alternative embodiment, is tocapture the actual received audio at the SHAD 114, and correlate it witha stored copy of the actual media audio provided to the audio system 112by the SH 110 (directly or indirectly via monitor 111). Correlatingthese representations or the acoustic signals will allow for the delayestimation from the SH 110 to the SHAD 114, and in some embodiments fromeach of the individual speakers 113-1 to 113-4 based upon a knownspecific “orthogonalization” of the signals. For example, the left andright audio will generally have significant differences and may beorthogonalized per known signal processing techniques. The tweeter andwoofer audio will also have differing responses, including amplifiercross over effects. For known speaker sources such as Dolby® 5.1,specific signals are send-able to each speaker and control of thesesources is possible.

Modification of the source signals by phase modulation to aid in theacoustic sounding signal orthogonaliztion is also contemplated. Forexample, the phase modulation of acoustic signals is generally difficultto detect for the human ear. As a result, an embedded sounding signalderived from the delivered media, using an orthogonal code on it willallow for orthogonalizaiton of the simultaneously transmitted soundingsignals. When one speaker is sounded at a time, no suchorthogonalization between speaker outputs is required, though it mayremain beneficial for specific reasons. In one embodiment, the embeddedsounding signals, may be orthogonalized to the main media signal,allowing for superior detection.

FIG. 2 is a block diagram depicting an example architecture of a smarthub (SH) embodiment.

Referring now to FIG. 2, the Smart Hub 110 is equipped with a microphone217 and is configured to delay the primary video and audio to ensuresynchronized audio at the remote speakers 113. The microphone 217 isconfigured to “hear” the delay differences from the speakers anddelay-adjust the speakers. The Smart Hub 110 can perform roomequalization based upon deterred audio from integrated microphone 217,microphone array, or microphone in remote speakers 113, or in smartphone applications.

To effect HDMI port selection to output HDMI port, in one embodiment, aport processor 209 is utilized which may be a suitable HDMI switch, or amore sophisticated port processor allowing for audio and videoprocessing and the decoding of the content and re-encoding of themodifier content, such as utilizing silicon image port processorSiI9535, or SiI9533 by Silicon Image (www.Siliconimage.com), and mayincorporate audio processor 208 and video processor 215 within thecomponent. In one embodiment, the processor 207 is an Arm cortex basedprocessor running Linux® such as the ST SPEAr310 or other appropriateprocessor such as those from Texas Instruments (TI)®. In one embodimentthe Bluetooth module 201 including a TI CC2554 BTLE chip including an8051 programmable microcontroller, or the like. In one embodiment theWi-Fi is provided by Qualcomm® or Broadcom®, and may incorporate an armprocessor performing the functionality of processor 207, as well asBluetooth module 201, and other functional blocks such as memory 218.Not shown in FIG. 2 is nonvolatile flash memory, which may beincorporated within memory 218 in specific embodiments.

Port processor 209 provides the content form ports 1 to port n asselected to one or more output ports including output port 220 in theillustrated embodiment. Ports 219 and 220 may be capable of input for“return path” information in some embodiments (reverse audio forexample). Such parts as described may be capable of Mobile HighDefinition (MHL®) input or output in specific embodiments. Portprocessor and audio and video processor 208 and 215 are capable of audioequalization of the source ports to the output port(s) 220, 219 in someembodiments. In some embodiments, Wi-Fi interface 202 may receivestreaming content which may be provided to the port processor directlyor via processor 207. In some embodiments, one or more Universal SerialBus (USB®) interfaces 204 and 205 may receive streaming or storedcontent which may be provided to the port processor directly or viaprocessor 207. NFC interface 213 may be used for the pairing a SHAD 114with a SH 110, utilizing processor 207 for control of the process.

Video Analysis to Determine Activity on a Source Port or Stream.

The Smart Hub 110 may perform video analysis on a plurality of sources107—109 to autonomously switch between the sources, in one embodiment.By using video analysis to compare individual frames, the Smart Hubdetermines information concerning sources of content. The Smart Hub 110is capable of using static processing to compare images at differenttimes to determine content. The Smart Hub 110 will compare two images,determine if the image is changing (pattern matching over time), andautonomously switch to the most preferred source determined by theprobability processor, or a logical process such as those described herein and associated with FIG. 8. If change of content is detected, theSmart Hub may change sources to the active source, according to specificrules and processes (for example see FIG. 8). If the source has beenstagnant for a long period of time before someone is the room and aperson walks into the room, the Smart Hub would switch a newly activeport in one embodiment, as any newly active source being correlated witha newly arrived user, and a long period of inactivity would indicate ahigh probability of the newly present user having made the sourceactive, and desiring to view that active content. Some embodiments ofsuch logical flows are further described in connection with FIG. 8.

Audio Processing to Detect Active Content on a Source Port

The Smart Hub 110 may perform audio processing to determine if sound haschanged over time, or if the audio of a specific port is not present, orvery low level, or otherwise have change statistical level (generalvolume or dynamic range for example). The Smart Hub may listen forchanges is electrical sound waves from the plurality of sources andautonomously switch between sources when a one source changes from aninactive state to an active state. Such processing may also be utilizedfor the detection of commercial associated with times of increasedvolume, or other properties of the audio, such as a sample histogram ordynamic range which may be unique to a specific content providerindicated an alternates media (show vs commercial). The reasons tochange selected source may be different under differing conditions. TheSmart Hub may detect user input from one of the multitude of activesources and autonomously switch to that source. Smart hub may also useSmart combinations of events with monitoring content and correlatingthem to determine what to do. For example, if someone has been presentfor a long period of time and a source has been playing for a longperiod of time, it may be a poor time to switch sources when a new userenters the room. Smart hub 110 may use metadata to determine when toswitch, such as movie length metadata or commercials. Smart Hub 110 mayalso activate picture in picture during commercials.

FIG. 3 is a block diagram depicting an example architecture of anembodiment of a smart hub remote control (on a smart hub applicationdevice SHAD embodiment).

Referring now to FIG. 3, in one embodiment, a remote 312 can be used tocontrol the Smart Hub. The remote 312 can be integrated with WiFi,Bluetooth, etc., to communicate with the Smart Hub 110. The Remote 312has the capability to perform “local equalization”, in one embodiment,of the speakers within the room as discussed. Local equalizationequalizes the volume of a plurality of speakers around the remote tocreate an optimally pleasant experience for a user sitting next to theremote (or the SHAD 114). Further, the Remote can be used to equalizethe audio of a plurality of speakers located within the room to createoptimal acoustic characteristics within the room (real time roomequalization and can be inaudible to the listener). The Remote 312 maybe any wireless remote device including a smart phone or tablet. Theremote may be a smart phone using an “app” configured to control theSmart Hub 110 and may be used control the Smart Hub 110, act as anspeaker for personal listening, or display second screen content. TheRemote App of a SHAD 114 can capture voice commands to switch betweensources, or perform other functions. The voice recognition could beperformed in the Smart Hub, Smart Phone, or cloud. Third partyapplications, such as SIRI®, may perform the voice recognition. SmartHub 110 can provide metadata for statistical data about what content iswatched and by whom. The App may display secondary information relatingto the content that is being displayed from the active source, such asadvertisements for products and the like.

The SH 110 in one embodiment is capable of performing the “switching”over the same physical interface, but between different content sources.For example, it may switch over Ethernet/WiFi from two differentstreaming sources. Such sources could be Netflix, Hulu, Amazon Primevideo, a DVR with a digital interface, etc.

The server SHCS 103 may learn and update settings on the Smart Hub basedon information from the Smart Hub or from a user's device, or a webportal associated with the service, and/or by sending switch signals toa content providing device over the HDMI upstream link and/or byrequesting and receiving channel guide information from the user'sdevice.

The server SHCS 103 may interact with a consumer device based uponcollected information from the Smart Hub (SH) 110. The server 103 mayperform pushing of information directly, or indirectly, for example viathe Smart Hub. The server 103 may configure the Smart Hub 110. Theserver may learn and update settings on the Smart Hub based oninformation from the Smart Hub 110 or from a user's device, or a webportal associated with the service. The server may determine whenspecific ports have been switched. The server, in one embodiment, mayprovide for payment/authorization and enablement of new services on thedevice (any other the service in the below may be non-conditionalservices as well). Authorization services may be provided for WiFicredential sharing to new devices wanting to come onto the network.

Additional services provided by either the SH 110 and/or the SHCS 103may include content filtering and not allowing for switching to thatcontent (parental rules/controls), audio processing (audio “roomequalization”), providing display services, picture in picture (PIP),“commercial identification”, and switching back when the commercial isover. The SH 110 may enable secondary show or “standby” show watchingduring commercials or may invoke a DVR input during an identified period(such as before a show starts, or during a commercial). The SH 110 mayswitch to the DVR input, and control the DVR to play during the durationof the commercial. It may also be arranged for: Keeping a PIP of theprimary show on the screen to allow for manual switch back; keeping aprimary DVR of short period of time, to allow for correction of mistakesin switchback to primary; or catching up to real time during nextcommercial period. For example: SH 110 may be playing the Nintendo Wiiduring the commercial period or may be switching to social media orother related content—similar to the mobile device “second screen”display for the commercial period times. This is thus providing for adisplay “overlay” at the control of the Smart Hub, and added to theselected input (for commercial switch for example) or social interactiondisplay such as Twitter trending or for directions to associate a deviceor for configuration.

Based upon any event or condition, Smart Hub may also start or stop thetransmission of iBeacons (or other beacons) at the Smart Hub. Forexample, if the Smart Hub does not have a clear switch decision, it caninvoke the Smart Hub control application on the user's device, but notinvoke it when it is not required. Multiple invokable screens on theuser's device are also possible including: Switcher control, secondscreen services, configuration service, cloud interface, serviceinterface, advertising interface, offers interface, and/or incentivesinterface. In one example embodiment, the server can manage a loyaltyprogram for an account associated with the Smart Hub and or one or moremobile devices or an individual or group of individuals.

The smart hub 110 may mute or switch over to another source for theduration of commercials and other events. It may add other audio sourceswitchover as a secondary audio input during commercial, such as musicfrom other inputs, for example streaming from the internet, form otherHDMI inputs, Airplay, uPnP, other wireless sources (BT, WiFi), otherinputs, optical, RCA®, audio jack, et. The Server may stream audio fromPandora®, Spotify®, etc. The SH 110 would then perform video or audioanalysis based upon past statistics such as audio levels to determinethe commercial time periods (or other time periods of interest) andbased upon input designating a commercial or a show period, analyzingcollected information to refine the commercial or “secondary” period oftime. The SH 110 will switch back to the primary source once it hasdetermined that the commercial is over. For example: storing a runningaverage audio level, and upon input “this is a commercial”, SH 110 candetermine the interval over which the period is consistent based uponthe audio or other ongoing information (images, blackouts, etc.), the SHcan then categorize statistics related to the time periods and for timeperiods not designated as commercial (or designated as not commercial)and set up, or refine the rest of the audio periods.

Audio Leveling and Filtering Feature

The SH 110 may provide, in one embodiment, audio “leveling” (i.e. noexcessive loud or soft sounds) using an audio Automatic Gain Control(AGC) approach, and/or aid in the approach for users with some hearingloss, and amplify the audio, but dynamically adjust it, so as to preventexcessive volume deviations. In one example, the SH 10 may a spouse frombeing woken up, filtering excessively loud explosions, etc.). In otherexamples, the SH 100 may identify specific types of sounds such as thosenot suitable for children—curse words/profanity/explicit, and lowervolume during those short periods, or the server may blank video duringinappropriate scenes having been detected, or switch to another source,or modify the image so as to “cover” inappropriate images (automaticcensoring).

Content Filtering and Smart Hub Bypass Prevention

The SH 110 may physically lock the device to the HDMI input in oneembodiment, detect the rating or other aspect of the content and preventit from being displayed. The SH 110 may display a message, or switch toother content, or blank screen and display a message, potentially withpicture in picture. Alternatively the SH 110 may sense when the HDMI orother port has been disconnected from the display 111, and alert thedesignated parent that content may be played which is not authorizedthrough the SH 110. Such an alert may be provided via email, SMS,Twitter, or phone call, of other communications approach such asBluetooth. In one embodiment the Smart Hub 110 includes a microphone,which listens to the output of monitor 111, to verify that the contenton the monitor is the content that is being displayed, and providesalerts such is not the case such as when other content is displayed orplayed on the monitor other than the selected content form the SH 110.Further, a wireless or wired camera may interfaced with the SH toperform visual verification that the content displayed on the monitor111 is the content selected by the SH 110. In one embodiment, serverSHCS 103 provides messaging alerts when the smart hub 110 is notresponsive due to power being off, or other conditions, or alertsgenerated by the smart hub itself.

WiFi or Local Network Access Facilitator

The smart hub 110 may act as a WiFi “security” guardian using Bluetoothto connect a device, or to a WiFi Access Point (AP) signal. The“security” guardian feature allows a device to request connection to theWiFi network, using the app on the device to configure WiFi for thevisiting device (enterprise or regular security) with guest access tolocal resources, or other levels of access. In this instance, the devicepasses credentials to the visiting device via Bluetooth or via WiFidirect, or by means of beacons.

Time/Event Based Triggering of Universal Remote Combinations

In one embodiment, the smart hub 110 alone or in combination with serverSHCS 103 are capable or performing time/event based triggering ofuniversal remote combinations and sequences of controlling otherdevices. For instance alarm clock application turning the television onat a certain time, and switching to a specific program to wake a userup, or alert bed time turning all the electronics off, following awarning message displayed on the monitor 111.

Streaming of Audio, and Video Content from Smart Hub to SHAD 114

In one embodiment, the SH 110 is capable of providing selected inputsources (both streamed sources, or selected HDMI input sources, or othersources) over the Bluetooth or WiFi network for display on SHAD 114. Inone embodiment, such content may be delivered to the SHAD 114 utilizingcellular communications via a mobile network tower and base station 101.In at least one embodiment the content displayed on monitor 111, andstreamed to SHAD 114 are the same and synchronized in time by comparingthe audio recorded at SHAD 114 from the audio system 112, and adaptingdelays in the SH 110 so that video and audio between monitor 111 andSHAD 114 are aligned in time to an acceptable tolerance. Such a delayadjustment is generally required prior to the output of the HDMI (orother) signal to the monitor, but after the selection of the contentstream for transmission to the SHAD utilizing application link 118, asthe streaming process generally incus more latency than the displayprocess to the monitor 111. In other embodiments, or configurations, theselection of content to stream to 114 over link 118 from the SH 110, maybe different form the content displayed to monitor 111, and come fromdiffering selected content sources and stream input sources, or othersources. For example a user of SHAD 114 may watch a show from source 1,where the monitor 11 and audio system 112 display and play content fromsource 2, or from another source such as apple TV, or a DVR. Oneapproach for transmitting the audiovisual content to the device is toutilize the MHL standard for a wired connection, as defined andassociated with the HDMI specification.

Probability Processor

Associated with the SH 110 is a processor that permits selection ofsources for fuzzy decisions as to making a content source switch. Forexample, if low probability of a switch exists, it can ask forconfirmation, if higher probability, it may make a switch with nofurther input. For example, if one is watching content currently, andonly one device is turned on, the system can display that content on thedisplay with no further input.

This can also use even additive values relative to a threshold todetermine a likelihood of a specific content source and then confirmwith the user the most likely source to switch to. For example, a usermay be identified as always watching cable, and never playing Wii.Weightings would be adjusted for that user and calculated on a persource basis. If the ‘top” probability exceeds a threshold, and is abovethe other probabilities by a defined amount, the switch is made with nofurther input. If too many sources have similar additive values, thesystem may ask the user to select from two or three most probablesources. In additional embodiment, a determination of a switch “event”should be evaluated via weightings as well, with various factors addingor subtracting differing amounts and compared to a threshold (one ormore thresholds) and or ratios of other factors or values.

FIG. 4 is an example software/firmware architecture functionalpartitioning diagram of the processing performed in an embodiment of asmart hub.

Referring now to FIG. 4, an example software and/or firmwarearchitecture of an embodiment of a smart hub 110 is depicted. In thepresent diagram no interconnections between the modules are shown,however interconnections between any and all of the depicted functionalmodules is contemplated. In specific embodiments, a subset ofinterconnections may be present to effect specific functional processingand data flows. Some specific modules related to hardware or softwaredrivers or other interfaces or required functional support modules maynot be shown but are contemplated and omitted for the sake of clarity,since such modules are commonly known in the industry to one skilled inthe art. It is further contemplated that embodiments of one or more ofthe depicted or implied modules may be combined together to form otherfunctional modules, or described or implied modules may be partitionedinto modules of more granularity, or both. Regardless, the specificfunctional partitioning of the depicted embodiment is not intended to belimiting, rather to be illustrative and other equivalent partitioningare in the scope of embodiments, having the same, more, or fewerfunctional modules and including all or a portion of the describedfunctional modules in specific embodiments. The responsibilities andembodied functions of the specific depicted modules are well understoodto one skilled in the art, based upon the names of the modulesthemselves, in view of the examples descriptions of specific embodimentselsewhere in this disclosure. Therefore, only specific modules will bedescribed as useful in the description of specific embodiments. Forexample, the parental control processing module 436 provides themanagement and enforcement of parent control policy, preventing contentfrom being displayed or otherwise played which a parent has configuredas being unsuitable for their child's consumption, or exposure. Suchenforcement may include the prevention of the playing of music, video,or images or may provide for censorship (skipping over portions,blurring portions, blanking out portions, blanking, or “beeping” overportions of audio or video as appropriate).

Module 401 depicts an operating system providing an operatingenvironment for other functional modules, such as the well known Linuxkernel or the like. Module 402 depicts an executive control, process andprocedures (ECPP) module which provides for the “top level” executableprogram, managing other processes and functional modules. The ECEPP, inone embodiment, is responsible for “stringing” together and drawing onthe interfaces, resources, functional capabilities, state management(storage of conditional states for example), and the procedural flowsdescribed by a plurality of other functional modules so as to achievethe top level desired behavior of the smart hub (or other embodiment)desired capabilities and functionality.

403 depicts a user present/not present processing module which performsthe decision making process associated with when a user is present inthe room as detected by processing described elsewhere, or the decisionand state of when no users are present in the room.

404 depicts a display/monitor processing module, which provides theconfiguration and control associated with the port processor so as toeffect the display utilizing monitor port 219 to monitor 111. Forexample, the invocation of picture in picture (PIP), displayed contentgenerated or formatted locally, configuration and management of settingsrelated to (directly or indirectly the video processor 215, and in somecases audio processor 208.

405 depicts a device location processing module, which provides theprocessing required to identify the location of a device, associatedwith a user in one embodiment, as being present in the room, or notwithin the room, or to determine a relative distance from the SH 110,monitor 111, or speakers 113-1 to 113-4, and to utilize this informationfor the location, proximity, or presence on the room, or a region of theroom determination. For example, the determination that a user's deviceis located in the region in the viewable area of monitor 111 may beutilized for consideration in other modules and processes. In anotherexample, successive location determinations indicate the user (the SHADdevice) is consistently moving about the room, may indication that useris not watching the video content and under some conditions may resultin ignoring their presence in the switch recommendations, or alternatesuggestions of content of more interest, or a suggestion to switch to anaudio only program, or in one embodiment, a pause of the content whenthe users has left the room (in association of the Multiple users/singleuser present processing module 427) in on embodiment. In someembodiments this module (405) may also perform or be associated withperforming the room audio equalization processing which may be jointlyperformed to determine the range of the SHAD form each speaker so as todetermine, or otherwise estimate location. Such processing may beperformed in cooperation or coordination with the audio processingmodule 453, and the Sonic Transmission and reception module 452, and theconnection management module 448, and communication module 433, amongothers.

406 is a media identification module, in one embodiment, which interactswith a server to provide a sample of audio of one or more sources to anaudio finger printing service reachable via the internet cloud 102, andreceive meta data associated with the identity of the provide audio.Such audio may be extracted form a movie or other video program source,or a game, or the like. The identified content identity in someembodiments may be passed to the calling module or function (forinstance the ECPP 403), which in one embodiment may also call theso-called “related content identification module” 422, directly orindirectly.

407 depicts a “smart hub” cloud server interaction module which providesfor the protocols credentials and processes used for interacting theSHCS server providing specific APIs and capabilities in one embodiment.

408 depicts a cloud server interaction module which provides forinteraction with generic could services or servers and the protocols andprocesses associated with these interactions.

409 depicts a management module which is useable in some embodiments forconfiguration and control of the device.

410 depicts a user interface module which provides for user specificgenerations of displaying useful for requesting and displayinginformation to a user, on the monitor 111, via buttons or otherinput/output (I/O) on the SH 110, or on a SHAD 114.

411 depicts a user located/not located processing module which performsprocedures based upon if a user has been location or has not beenlocation and follows differing processes depending upon such location.This module may additionally invoke the location processing to attemptto location a user device.

412 depicts a HDMI/MHL processing module.

413 depicts a switch event identification module which determines when aswitch event is likely desired, and depending upon implementation mayattempt to identify the specific source to switch to, though this may beperformed elsewhere in other embodiments, such as in the switch sourceidentification module 421 in one embodiment. Such an embodiment issimilar to that depicted in flow diagram of FIG. 8.

414 depicts a commercial identification module which provides for thedetermination of periods of time which are likely to be commercialsbased upon the above discussed approaches, as well as others such asdetermining that commercials are likely to be presented as they have notbeen detected for a period of time, or that the identification of acommercials is not likely to be detected as the currently viewed contentrecently was identified as returning form commercial within a period oftime making another commercial less likely. Such “likelihood”estimations may be utilized to adjust the requirements for detectingcommercials or the period over which (how quickly) a commercial may beidentified.

Gesture Input and Menu navigation and inputs—Invocation of SHADapplication via beacons—module 414.

This module 414 (or another module, such as 415 application interactionmodule, 448 connection management module, communication module 433, ordevice discovery module 434) may also be responsible in someembodiments, for requesting confirmation from one or more users on theirSHAD devices, or via a notification on monitor 111, for a suggestionaction (such as that a commercial may be likely and requesting input toswitch to secondary content in this example). Such input may come fromuser input on a SHAD device (tapping, shaking, “nodding” the device yes,shaking or twisting the device “no”, or the like utilizing the sensorson the device, a input on the device screen, a audio input on the devicemic, or the SH mic, or the like using “clap” detection, voicerecognition, or the like). Such described inputs and gestures may beconsidered as usable for input associated with any confirmationassociated with any step whether explicit conformation is mentioned asbeing performed or not in other sections of this disclosure, and may beperformed in other modules and processes and stages as well. Furthersuch gestures and inputs may be usable in general screen menu navigationas well. Such gesture confirmations may be enabled by the SHAD and/orhub 110 generating additional or modified beacon UUIDs or major and orminor values within beacons so as to cause a device to invoke the SHADapplication into at least the background state so as to measure thegestures and inputs, and send messages back to the SH 110 confirming ornot confirmation an action or menus navigation process.

415 depicts a mobile application/mobile device interaction moduleinterfaces to other devices allowing for control and second screencontent. Some embodiments may utilize this module for interacting withSHURE devices (such as 120) as well.

416 depicts an SIP user agent and associated processing (SUAAP) moduleusable for IP video sources, VoiP conversations, IMS serveries(3GGP.org), VoiP services, presence services, chat and IM services,video conferencing services, and the like.

417 depicts a memory and storage management module for Flash and RandomAccess Memory (RAM) 218 interaction, and other storage media such as aUSB flash drive utilizing USB port 204 or 205, a hard drive (such as203).

418 depicts a user profile module which stores details and SH 110behavior preferences, and user behavior trends for each identifier user,and relating to a specific user past behavior, selections, preferences,which may be used to provide content on the second screen SHAD 114, themonitor 111, content switching preferences in SH 110, advertisements,offers, personally private content, content viewing permissions,parental control authority over other user profiles, parental controlpermissions, SHAD 114 association of a device profile with the specificuser, content preferences, content source preferences etc.

419 depicts a user identified/not identified processing module which inone embodiment provides for the local decision process associated withSH 110 content switching prediction when users are or are notidentified, relating to their preferences, priorities, and profiles frommodule user profile module 418.

420 depicts a boot control/firmware update control module

421 depicts a switch source identification module which in oneembodiment utilizes an interface associated with a switch input such asutilizing HDMI 1.4 HDMI Ethernet Channel (HEC) to enable IP-basedapplications and interfaces, or other interfaces such as withThunderbolt®, USB, or the like. Such interfaces identify informationrelating to the content being provided by the source, other availablecontent form the source, a channel guide from a cable box for example,other DVR recoded shows, or the equipment providing such contentutilizing the source interface. This module may work with the mediaidentification module 406 in some embodiments, and with the SHCS cloudserver 103 either directly or indirectly to aid in the determination ofthe identity of the source equipment and interfacing capabilities (forexample, utilizing the HDMI or MHL protocol specifications and derivedinformation). Information determined utilizing the HDMI Ethernet Channel(HEC) interface on one or more HDMI 1.4 ports may be stored in theappropriate device profile in the device profile module 426. In someembodiments this module 421 may alternatively or in addition identifythe source to which a switch of content should occur in association withthe switch event identification module 413. Such an embodiment issimilar to that depicted in flow diagram of FIG. 8.

422 depicts a related content identification module which operatesalone, or with other modules, and in some embodiments interfacing withthe SHCS cloud server, 103 either directly or indirectly (via Cloudserver interaction module 408, or SHCS interaction module 407, forexample in specific embodiments) and one or more of the applicationplatform module 437 and managed platform services 438-463, to determineand acquire content relevant to an identified content. For example, itmay be determine tweets relating to a specific hash tag related to acurrently viewed/displayed show on monitor 111. This module in someembodiments may be involved in the determination of specificadvertisements from sponsored advertisement module 432, offers orrewards related to the displayed content from the LCRS Module 431, andthe like. The content identified may be utilized and provided to thesecond screen interaction module 424 for further processing andpotentially display on the SHAD 114, or another SHAD. The content mayadditionally be used by other modules for display as an overlay on themain monitor 111, or as a picture in picture with the content, forexample, displayed in the secondary video box. The related content mayfurther be utilized during periods of identified commercials to providecontent relating to the currently viewed program, and avoid directlyviewing the commercials. Such content may be displayed as the onlydisplayed content during commercial periods, primary content, with thecommercial in the secondary PIP (picture in picture) window, or thelike.

423 depicts a local server interaction module for, in one embodiment,retrieving and playing content forma local networked drive or folder, orinterfacing with a local shared library from iTunes, or interacting witha local media server.

424 depicts a second screen interaction module which determines thecoordination of content currently, or recently being viewed on themonitor 111, and provides for screen display formatting considerations,and presentation of content in associated with the MobileApplication/Mobile device Interaction Module 415.

425 depicts a hardware I/O module providing abstraction for the hardwarefrom the other modules.

426 depicts a device profile module which provides for information whichis known about the smart hub 110 itself with regard to configuration andsettings, as well as other devices, for example, the details associatedwith the SHCS 103, one or more SHAD 114 devices, and associatedapplication information, audio system 112, “other controllable device”140, Source 1 (140) through Source n (109), and SHURE devices (120).Such known information, in embodiments, may include the type of device,manufacturer for the device, model of the device, SN, capabilities,communications capabilities and protocols (BT variant (BTLE forexample), WiFi variant, beacon capability and (UUID etc.), IR, IrDA,UDP/TCP/IP, MAC Address, SNMP, RTP, SIP, and the like), command sets,menu tree information, known stored content on the device,controllability and process of control of the device, encryption keys,passwords, usernames, credentials, etc.

427 depicts a multiple users/single users present processing moduleproviding for decision flow, in one embodiment, when for exampledetermining if and how the content the smart hub has selected forpresentation on display 111 is to be switched, when a plurality of usersare present. In such a condition, each user may be a differing predictedpreference for which content to play, and may have differing “priority”in consideration of the preference or their inputs by the smart hub.

428 depicts a universal remote control/local or connected device control(UDCM) module which in some embodiments provides for the translation ofcommands received from another sources (as IP packet application levelmessages for example form a SHAD (such as SHAD 114) and translates thosecommands or “macro” commands into one or more device appropriate IR orRF remote control commands to one or more devices to achieve a desiredobjective. Such a process may, in some embodiments, include the use offeedback from the controlled devices to trigger specific repeatedcommands, or modified commands or command combinations. Such feedbackmay be achieved by monitoring a switch port interface (210, 212, orother source) for activity on audio or video content as described. Sucha processes in one embodiment may further involve the utilization ofsource equipment interaction via communication channels utilizingEthernet (including HEC as discussed), WiFi, Bluetooth, or the like. Forexample identifying a cable box is powered on by utilizing the HECinterface to ping specific addresses, or discover default gatewayaddresses for example.

429 depicts a home network security, policy, and guest (HNSPG)processing module which provides for the authorization and admittance ofa guest and other users onto the a home or local network of which the SH110 may be directly providing acting as a WiFi access point, or as anagent to another access point, able to interact with that wirelessrouter to allow admittance/authorization, such as a RADIUS serverinterface to a WiFi router using EAP protocols as defined in the 802.11and related standards. Such a process is described as an example inforegoing sections. In one embodiment, the SH 110 may act as a truelayer 2 access point with no routing, relaying on a home network routerit connected to the SH 110 as a DHCP server, and acting as a layer 2bridge to the admitted devices. In other embodiments, the SH 110 mayshare credentials with the device for access to the network via anapplication on that device, or utilizing SMS, Bluetooth (BT), AirDrop®,or the like. In one embodiment, the SH acts as a radio authenticationserver to a home WiFi router, as mentioned, and configures the WiFirouter utilizing Simple Network Management Protocol (SMNP) andstandardized interfaces automatically utilizing the management interfacepassword to configure the Access Point (AP) to perform enterprisesecurity procedures with the SH 110 populating the credentials, basedupon various factors, such as the device Media Access Control (MAC)address of the device to be admitted. The device MAC address, iscommunicated to the SH 110 via an application on the guest device, andadmittance optionally approved by a SHAD 114 device in the process. Asmall application to send the MAC address to the SH may be provided tothe guest device form an internet URL, or “Airdropped” from an IOS®device to another IOS device, or the like. Device discovery module 434in one embodiment would then detect the guest device running the guestapplication (via BT beacons detected by the guest device, andestablished communications to the SH for the guest, or the SH detectingbeacons form the guest application, or another process using wirelesscommunication), passing the MAC address and other information in someembodiments, (users name, or device name, or other identifyinginformation, a picture of the user taken by the application, etc.) whichmay be forwarded to a SHAD device such as SHAD 114 for approval. Uponapproval the credentials are authorized within the AP (internal orexternal to the SH 110), and instructions are sent back to the guestapplication instructing them to select a specific network WiFi SSID, oris performed by the application itself automatically. Alternativelyguest access may be provided via BT, or Bluetooth High-speed, or WiFidirect, and bridged to the local network.

430 depicts a decoding, encoding module (MP3, MP4, MPEG, AAC, etc.).

431 depicts a loyalty, coupons, rewards and stored value (LCRS) module.

432 depicts a sponsored advertisement module which may retrieve aninventory of sponsored advertisements from an ad server prior to or inreaction to a request, which may be based upon the identification ofcurrently being viewed content (determined by media identificationmodule 406 in one embodiment).

433 depicts a communications module.

434 depicts a device discovery module which may perform beacon oriBeacon detection, or beacon or iBeacon generation with one ore moreUUIDs, or other identifiers. This module may also perform WiFi networkdiscovery or the discovery of other communications networks, orconfigure the SH 110 to transmit detectable WiFi beacons for Wi-Fidirect, or to act as a Wi-Fi AP, or other communications protocol.Further, the module may support the Bonjour protocol or the like for thedetection of two other devices on a location area network, or toadvertise services available on the LAN.

435 depicts a stream reception and processing module provides for thereception of stream content which in some embodiments may includeservice such as Netflix, or the like, or streamed content form a UPnPdevice, or AirPlay, Chromecast or the like.

448 depicts a connection management module providing for the setting upand management of specific communications connections and any handoverwhich may occur.

449 depicts a security authentication authorization (SAA) module whichprovides an alternative and more comprehensive embodiment of the processdescribed in module 429, including intruder detection and “per role”access to network resources and content.

450 depicts a stream to remote device processing module which acts toallow the streaming of audio and/or video to a remote device which asfor display on a SHAD 114, or playback on the headset jack of SHAD 114,and to aid in the timing synchronization of the audio played on the SHADwith the video displayed on the monitor 111, or on the SHAD screen, asdiscussed elsewhere utilizing time delay and equalization. In oneexample, the delay of the replayed audio to the audio jack of the SHAD114 is measured as discussed, and the video the monitor 111 is delayedso as to synchronize the audio with the video. Other sources such asaudio form device 112 may be synchronized as well, and utilizeadditional modules such as the audio processing module 453, sonictransmission and reception module 452, using procedures discussedpreviously.

451 depicts a general processing module.

452 depicts a sonic transmission and reception module which provides, insome embodiments, for the generation of audio waveforms which may beused to perform “ranging”, sounding, equalization/channel estimation,and location processes as discussed, and may utilize approaches whichprovide for non-detectable signals such as those embedded within anaudio signal, a phase modulated signal, a high frequency signals notdetectable by the human ear, or the like. Other embodiments may generatesignals which are detectable which may be sent a low levels, for longerdurations in time, or only occasionally. Some waveforms were previouslydiscussed. Module 452 may in the same or other embodiments, provide forthe reception and processing of such signals, with transferred fromanother device that device having received the audio signal, or receivedin the SH 110 directly. Other embodiments may provide for the waveformgenerated to be passed to other devices such as a SHAD 114 forprocessing associated with reception at that device.

453 depicts an audio processing module which in various embodimentsperforms audio equalization, dynamic volume control, and the like addiscussed elsewhere.

437 depicts an application platform module which allows for thedownloading of third-party applications, or simply downloadableapplications to operate in a protected area, in one embodiment. Suchapplications may allow for unique or proprietary/confidential credentialstorage, protocols, transcoders, menu applications, and the like.Platform applications would be developed in one embodiment with aspecific API to framework which interfaces with the Application platformmodule and isolate the platform applications form underlying changes,and enforces policy and security. The following are some exampleplatform applications, which in other embodiments may be so-callednative applications:

438 depicts Google services interaction.

439 depicts Microsoft services interaction module.

440 depicts Twitter interaction module.

441 depicts Shazam interaction module.

442 depicts Hulu interaction module.

443 depicts Yahoo interaction module.

444 depicts Lastpass interaction module.

445 depicts Facebook interaction module.

446 depicts Amex interaction module.

447 depicts Amazon interaction module.

454 depicts Netflix interaction module.

455 depicts Skype interaction module.

456 depicts Pandora interaction module.

457 depicts Square interaction module.

458 depicts Paypal interaction module.

459 depicts other interaction modules.

460 depicts Chromecast interaction module.

461 depicts iCloud interaction module.

462 depicts BOA interaction module.

463 depicts Visa interaction module.

FIG. 5 is an example software/firmware architecture functionalpartitioning diagram of the processing performed in an embodiment of amobile device smart hub remote control/interface application (residenton an embodiment of a smart hub application device—SHAD).

Referring now to FIG. 5, example software and/or firmware architecturesof an embodiment of a mobile remote application residing on a SHADdevice 114 are depicted. In the present diagram no interconnectionsbetween the modules are shown, however interconnections between any andall of the depicted or assumed and not depicted functional modules iscontemplated. In specific embodiments, a subset of interconnections maybe present to effect specific functional processing and data flows. Somespecific modules related to hardware or software drivers or otherinterfaces or required functional support modules may not be shown butare contemplated and omitted for the sake of clarity, as such modulesare commonly known in the industry to one skilled in the art. It isfurther contemplated that embodiments of one or more of the depicted orimplied modules may be combined together to form other functionalmodules, or described or implied modules may be partitioned into modulesof more granularity, or both. Regardless, the specific functionalpartitioning of the depicted embodiment is not intended to be limiting,but rather to be illustrative and other equivalent partitioning(s) arein the scope of embodiments described herein, having the same, more, orfewer functional modules and including all or a portion of the describedfunctional modules in specific embodiments. The responsibilities andembodied functions of the specific depicted modules are well understoodto one skilled in the art, based upon the names of the modulesthemselves, in view of the examples descriptions of specific embodimentselsewhere in this disclosure. Therefore, only specific modules will bedescribed as useful in the description of specific embodiments. To theextent the names of the specific modules of FIG. 5 are similar to thoseof FIG. 4, in some embodiments these modules would act in a similarmanner. In other embodiments, and where appropriate, similarly namedmodules would act as a counterpart to the modules on the SH 110 and tointeract with their counterparts so as to achieve a specific objective,in a complimentary approach.

Mobile Remote Application Processing (FIG. 5)

501 depicts an OS interface module that provides for interfacing to theoperating system of the SHAD 114 device, such as interfacing withandroid, IOS, Linux, or the like.

502 depicts an executive control processes and procedures (ECPP) module.

503 depicts headset interface processing which is received from, in oneembodiment, the stream to remote device processing module 450 (eitherdirectly or indirectly).

504 depicts a camera interaction and processing module.

505 depicts device location processing which in some embodiments worksin coordination with location processing on the SH 110 and locationprocessing module 405, sonic transmission and reception module 558, andother modules.

506 depicts a media identification module.

507 depicts a “Smart hub” cloud server interaction module.

508 depicts a cloud server interaction module.

509 depicts a management module.

510 depicts a user interface module.

511 depicts a user located/not located processing module.

512 depicts an NFC pairing and interaction module.

513 depicts an event identification module.

514 depicts a smart hub interaction processing module.

515 depicts a mobile application/mobile device interaction module.

516 depicts an SIP user agent and associated processing (SUAAP) moduleprovides for a SIP protocol user agent functionality allowing for VoiPcalls, video and presence applications, as well as IMS related services.

517 depicts a user profile module.

518 depicts a user identified/processing module.

519 depicts a motion and orientation determination processing module.

520 depicts a related content identification module.

521 depicts a local server interaction module.

522 depicts second screen interaction module.

524 depicts a device profile module.

525 depicts a universal remote control/local or connected device control(UDCM) module which provides for interacting with the SH 110 using WiFiLAN messaging or BT communications so as to send instructions andreceive feedback to and from the associated universal remotecontrol/local or connected device control (UDCM) on the SH 428, and asdiscussed further elsewhere herein. Additionally 525 is capable, inspecific embodiments, of interacting with the SHURE 120 and like devicesallowing for universal IP and RF control from the SHAD device 114,either from the SH, or from the SHURE 120, or a combination of the two.

526 depicts a home network security, policy, and guest (HNSPG)processing module that provides complimentary processes to the SH 110,and the associated module as discussed.

527 depicts a decoding encoding module (MP3, MP4, MPEG, AAC, etc.).

528 depicts a loyalty coupons, rewards, and stored value (LCRS) module.

529 depicts a sponsored advertisement module.

530 depicts a communications module.

531 depicts a device discovery module.

532 depicts a stream reception and processing module.

533 depicts an application platform module.

534 depicts a Google services interaction module.

535 depicts a Microsoft services interaction module.

536 depicts a Twitter interaction module.

537 depicts a Shazam interaction module.

538 depicts a Hulu interaction module.

539 depicts aYahoo interaction module.

540 depicts a Lastpass interaction module.

541 depicts a Facebook interaction module.

542 depicts an Amex interaction module.

543 depicts an Amazon interaction module.

544 depicts a Netflix interaction module.

545 depicts a Skype interaction module.

546 depicts a Pandora interaction module.

547 depicts a Square interaction module.

548 depicts a PayPal interaction module.

549 depicts other interaction module(s).

550 depicts a Chromecast interaction module.

551 depicts an iCloud interaction module.

552 depicts a Bank of America (BOA) interaction module.

553 depicts a Visa interaction module.

554 depicts a connection management module.

555 depicts a security authentication authorization (SAA) module.

556 depicts a stream from Hub device processing module.

557 depicts a general processing module.

558 depicts a sonic transmission and reception module.

559 depicts an audio processing module.

FIG. 6 is an example software/firmware architecture functionalpartitioning diagram of the processing performed in an embodiment of amobile device smart hub server application.

Referring now to FIG. 6, example software and/or firmware architecturesof an embodiment of a smart hub cloud server (SHCS) 103 are depicted. Inthe present diagram no interconnections between the modules are shown,however interconnections between any and all of the depicted, or assumedand not depicted, functional modules is contemplated. In specificembodiments, a subset of interconnections may be present to effectspecific functional processing and data flows. Some specific modulesrelated to hardware or software drivers or other interfaces or requiredfunctional support modules may not be shown but are contemplated andomitted for the sake of clarity, as such modules are commonly known inthe industry to one skilled in the art. It is further contemplated thatembodiments of one or more of the depicted or implied modules may becombined together to form other functional modules, or described orimplied modules may be partitioned into modules of more granularity, orboth. Regardless, the specific functional partitioning of the depictedembodiment is not intended to be limiting, but rather to be illustrativeand other equivalent partitioning(s) are in the scope of embodimentsdescribed herein, having the same, more, or fewer functional modules andincluding all or a portion of the described functional modules inspecific embodiments. The responsibilities and embodied functions of thespecific depicted modules are well understood to one skilled in the art,based upon the names of the modules themselves, in view of the examplesdescriptions of specific embodiments elsewhere in this disclosure.Therefore, only specific modules will be described as useful in thedescription of specific embodiments.

For example if the switcher does not have a clear switch decision, itmay invoke the switcher control application on the user's device, butmay not invoke it when it is not required.

Smart Hub Server Application Processing (FIG. 6)

601 depicts an OS interface.

602 depicts an executive control, processes and procedures (ECPP)module.

603 depicts a boot control/FW update control module.

604 depicts a camera interaction and processing module.

605 depicts device location processing.

606 depicts an advertisement server interface.

607 depicts a merchant and advertising interface module.

608 depicts a smart hub interaction processing module.

609 depicts a management module.

610 depicts a device profile module.

611 depicts a demographics determination processing module.

613 depicts a behavior determination processing module.

614 depicts an advertisement database.

615 depicts a mobile application/mobile device interaction module.

616 depicts an SIP Proxy and associated processing (SUAAP) module.

617 depicts an employee web portal.

618 depicts a user profile module.

619 depicts a user identified/not identified processing module.

620 depicts a user interests determination processing module.

621 depicts a media identification module.

622 depicts a related content identification module.

623 depicts an advertisement determination module.

624 depicts a second screen interaction module.

625 depicts a customer web portal.

626 depicts a customer cloud services store module.

627 depicts a customer content store module.

628 depicts a universal device control (UDCM) interface module.

629 depicts a home network security, policy, and guest (HNSPG)processing module.

630 depicts a decoding encoding, trans-coding module.

631 depicts a sponsored advertisement module.

632 depicts a loyalty, coupons, rewards and stored value (LCRS) module.

633 depicts a communications module.

634 depicts a customer down-loadable app store module.

635 depicts a stream reception and processing module.

636 depicts a database management and interface.

637 depicts a user, device and smart hub related databases.

638 depicts application subscription databases.

639 depicts payment processing.

640 depicts a third party services interface.

641 depicts a connection management module.

642 depicts a security, authentication authorization (SAA) module.

643 depicts a streaming services transmission processing module.

644 depicts a general processing module.

645 depicts an application platform support module.

646 depicts a sonic transmission and reception module.

FIG. 7A is an example flow diagram of a smart hub application device(SHAD) embodiment detecting the presence of a smart hub (SH) or arelated devices (such as a smart but universal remote extender, or theanother device) an invoking an embodiment of a smart hub interfaceapplication.

FIG. 7B is an example system level flow diagram of a smart hub systemembodiment wherein a user with a smart hub application device (SHAD)embodiment comes into range of a smart hub embodiment to perform exampleservices.

FIG. 8 is an example system level flow diagram of a smart hub embodimentperforming a embodiment of a source switching process.

FIG. 9A is an example Illustration of SH system utilizing combinedAcoustic and RF Processing to determine information related to the SHADdevice embodiment.

FIG. 9B is an example system flow diagram a smart hub system processembodiment utilizing combined Acoustic and RF Processing for thedetermination of Acoustic channel state information (CSI) (one or moreof: presence in a specific area presence, location (proximal orabsolute), acoustic channel equalization, spatial equalization, and thelike).

FIG. 10 is an example block diagram of a Smart Hub Universal RemoteExtender (SHURE) 120 embodiment (allowing for a smart hub to provide IRcontrol (or other control) to controllable devices providing contentsources to the switchable ports of a smart hub, when such devices arenot directly controllable by resources internal to the smart hubembodiment components), in one embodiment.

As previously discussed above, associated with FIG. 10 is a UniversalRemote Extender 120, using the smart hub extender as a stand-alonedevice with an application on the smartphone to act as a universalremote control device.

Further details of the depicted modules and components are as follows:

1.1. Smart Hub

-   -   may be a “smart” HDMI switch    -   using video analysis to determine which stream to connect to    -   monitors audio and or video to determine what is being watched.    -   connects to home WiFi network and to the cloud    -   can “broadcast” audio to remote speakers    -   will delay the primary video and audio to ensure synchronized        audio at the remote speakers    -   may have a integrated speaker to “hear” the delay differences        and delay adjust them.    -   can perform “room equalization” based upon deterred auto from        integrated mic, mic array, or microphones in remote speakers, or        in smart phone applications    -   a smart phone application can be used to transmit signal from        the local smart phone to the cloud or HDMI box (a smart hub        which receives one or HDMI inputs or other local intermediary        device using some other equivalent interface) to perform        synchronization/equalization,    -   can also be used for “finger printing what is being watched.    -   mic and WiFi or BT can be integrated into a remote with the same        functionality as above.    -   can put a speaker in a couch pillow for example, or other object    -   can perform “local” equalization functions.    -   remote can be used to provide headset jack for TV audio, with        delay sync to video.    -   geometry of the room can be deterred as well.    -   send audio via WiFi between the phone app and the HDMI box, and        phone can provide feedback to equalize delay to smartphone app        such that video matches audio in terms of delay synchronization    -   can use phone speaker phone as well to augment the in room audio    -   use smartphone app to manually switch between HDMI connections        Equalization performed via inaudible embedded signals, realtime    -   look at embedded phase modulated signals    -   look at low level signals        also use finger printed and identified shows to send information        to the phone for marketing and advertisement purposes        can use the signal in the HDMI signal to determine what shows        are being watched        augments the video to display on screen information from the        cloud based upon what the phone app is doing and used input    -   is user configurable        -   1.1.1. Interacts with a special purpose remote device        -   1.1.2. Interacts with an application on a smart phone            (iPhone 5s, Android device, etc.)        -   1.1.2.1. transmits beacons or other indications (over            802.11, BT, BTLE, LAN, etc.)        -   1.1.2.2. may use Bonjour protocol over LAN        -   1.1.2.3. Causes the invocation of an application on a iPhone            (in one example) based upon the detection of an iBeacon            -   1.1.2.3.1. An app is loaded onto an iOS device and                registers a proximity UUID with the OS            -   1.1.2.3.2. upon detection of the iBeacon, the app is                invoked in the background            -   1.1.2.3.3. depending upon the scenario different actions                may be taken                -   1.1.2.3.3.1. a push notification may be sent to the                    user to allow them to bring the application to the                    fore ground                -   1.1.2.3.3.2. communications with the SH or the SH                    server may be established, renewed, or otherwise                    processed                -   1.1.2.3.3.3. spending upon the state of the                    interaction with the user of the device, different                    screens may be presented, and different actions                    taken by SH and/or SH Server                -   1.1.2.3.3.4. SH may attempt to location the user as:                -    1.1.2.3.3.4.1. being in the room using acoustics                -    1.1.2.3.3.4.2. location the user specifically                    within the room using acoustics                -    1.1.2.3.3.4.3. locate the user (the device) within                    the room using BTLE ranging to one or more devices                    in a coordinated approach                -    1.1.2.3.3.4.3.1. using SH remote extender beacon                    ranges reported form the application or vice versa                    for example                -    1.1.2.3.3.4.3.2. a combination of acoustic and BTLE                    approaches        -   1.1.3. May include “universal remote” capabilities            -   1.1.3.1. controls other devices using IR or RF remote                know remote control protocols                -   1.1.3.1.1. Can control other devices directly by IR                    or RF remote control standards                -    1.1.3.1.1.1. Can use BT or BTLE to control the                    other source devices if possible                -    1.1.3.1.1.2. Can use 802.3 or 802.11 variants to                    control the other devices                -    1.1.3.1.1.3. Can use IF (including IrDA standards)                -    1.1.3.1.1.4. Any other interfaces are contemplated                    as well.                -   1.1.3.1.2. Coordinate the universal remote control                    operation with the content source selection                    operation                -   1.1.3.1.3. Interface with “remote extender devices                    to control other items”

1.2. Can “Learn” Behaviors

-   -   After long pauses, which input get watched most often?        -   based upon time of day, when turned on.        -   “watch the inputs” and time of day, and other factors such            as which devices are detected in proximity.        -   which inputs have audio or video active (as a voltage, or            with “moving” content), and        -   only change when something has happened,        -   qualify an “event” as a duration in time as well, in some            embodiments.        -   learn which devices are “always active” and which ones start            moving before switching to it. i.e. past behavior of their            “pre-switch” condition—to increase the “probability” a            switch should occur.    -   use Bluetooth, or WiFi proximity of one or more devices to help        determine who is watching, and qualify the rules (or        probabilities of switching).    -   use “beacon” like features to transmit beacons from the device        to phones.        -   use this to invoke an app on the phone in “invisible” mode,            with no prevention to the user necessarily        -   use beacon to invoke a visible UI or other display to the            user.        -   use inputs on the phone to influence switching or switching            preferences.        -   use the touch screen    -   use “movement of the phone” to make the change.    -   use the “invoked” app on the phone to perform “second screen”        functions to socially interact with other users watching the        same content (or listening)        -   use twitter or Facebook, or other social site with the            second screen app.        -   take polls—        -   push or display offers and advertising        -   send information via twitter alone        -   send information via sms, iMessage, google messaging, and            the like.        -   send emails to the phone    -   only push content when the device is detected via iBeacon like        approaches.    -   forced automatic switch by turning an input device off, then        back on again    -   add simple input to the device to force a manual switch.        -   a button        -   a speaker input with qualified sound signature        -   “clapper” input.        -   speech recognition            -   implemented in the smart hum            -   implemented in the smart phone application            -   implemented in the SH server            -   in the application on the smartphone, utilizing system                resources such as Siri from Apple (to generate a test                message to the device, to return a message)            -   using gestures to control the innovate of the app or the                manual selection of the source port, other actions, or                confirmation of the suggested port (such as tapping the                phone to confirm switching from one source to another)        -   A message    -   BT message or signal    -   WiFi/ethernet carried message    -   a receiver SMS to a specific address of a plurality of addresses    -   for example . . . one SMS phone number, or messaging address, or        sip address, etc., iMessage, IM, IMCP, XMPP, yahoo messages,        google messenger, etc., address for each desired device to        switch to.    -   all the messages being received by the same device, the device        responsive to a plurality of addresses.    -   qualify the switch based upon it being sent from a specific        address, or device.    -   prioritize the from addresses, allowing for protection from        override, or to allow for an override of other inputs form other        addresses.    -   The preceding overage aspects, and authorization from address        aspects, but with a single destination address, but with content        determining the “switch to” device    -   message to device with contents “TV” or “1” or “DVR” where the        content is interpreted based upon one or more of:    -   a name associated with a port (hdmi port for example)    -   a number of the port.    -   1, 2, 3 . . . A, B, B . . .    -   including in the message a name associated with the port    -   the device storing that name, and the using the name in future        messages to allow for switching    -   first message switch (“1 TV”)    -   second message switch (“tv”)    -   third message switch (“Wii”)    -   generating the message from a phone using a Siri like interface,    -   “Siri, new message to xxx (switchie for example) TV”)    -   the type of port it is . . . message (HDMI), USB, MHT,    -   storing the names in the cloud or at a server, and generating a        standard message to the device.    -   the above messaging implementations, but received by a server        which modifies a message, or generates a new message to the        device to perform the desired function

1.3. A Server

which backs up the settings from the device, which is connected.

-   -   which receives “audio or video” samples and identifies the        content being watched    -   transporting the “samples” as coded information—reconstructible    -   encoding or hashing the content in a non-reconstrucable was so        as to allow for intensification, but not necessarily recreation    -   performing the identification of the shows or audio content        locally on the device based upon downloaded “signatures” to        hashed content information    -   performing the identification on the server    -   identifying the content based upon information encoded with tin        the content steam itself such as metadata, or information        packets, or digital channel guide information, or interacting        with the content providing devices itself (Ethernet, or HDMI        interaction)    -   performing the “switching” over the same physical interface, but        between difference constant sources . . . for example over        ethernet/wifi from two difference streaming sources. Netflix,        Hulu, amazon prime video or a DVR with a digital interface.    -   sending switch signals to a content providing device over HDMI        upstream link.    -   requesting channel guide information from the device and        receiving it.        -   interacts with a consumers device based pun collected            information form the switcher device.        -   performs the punching directly, or indirectly, for example            via the switcher device.        -   configures the switcher        -   learns and updates setting on the switcher based on            information from the switcher or from a users device, or a            web portal associated with the service        -   the device or the server determining when specific ports            have been switched.        -   provides for payment/authorization and enablement of new            services on the device (any other the service in the below            may be non-conditional services as well)    -   WiFi credential sharing to new devices wanting to come onto the        network for example    -   content filtering and not allowing for switching to that        content. (parental rules for example)    -   audio processing (audio “room equalization” for example    -   providing display services, picture in picture, “commercial        identification” and switching back when the commercial is over.    -   secondary show or “standby” show watching during commercials.    -   invoking a DVR input during an identified period (such as before        a show starts, or during a commercial)    -   switching to the DVR input, and controlling the DVR to play        during the duration of the commercial    -   keeping a PIP of the primary show on the screen to allow for        manual switch back.    -   primary DVR of a short period of time, to allow for correction        of mistakes in switchback to primary.    -   catchup for real time at the next commercial period    -   playing Wii or another video game during the commercial period.    -   switching to a “social media” ro other related content—similar        to the mobile device “second screen” display for the commercial        period times.    -   Providing for a display “overlay”    -   at the control of the switcher device, and added to the selected        input (for commercial switch for example)    -   for social interaction display—twitter trending    -   for directions to associate a device    -   for configuration . . .    -   Provide for    -   A platform in the switcher (or other) device which allows for        the downloading or enablement of specific service    -   based upon any event or condition . . . start or stop the        transmission of iBeacons (or other beacons) at the switcher        device, and    -   for example if the switcher does not have a clear switch        decision, invoke the switcher control application on the user's        device, but don't invoke it when it is not required.    -   have multiple invokable screen on the users device.        -   switcher control        -   second screen services        -   configuration service        -   cloud interface        -   service interface        -   advertising interface        -   offers interface        -   incentives interface            add loyalty to an account associated with the switcher and            or one or more mobile devices or an individual or group of            individuals    -   tweet or other input on the mobile device, get points, or        rewards redeem points for rewards such as        -   product coupons, discounts,        -   enhanced services on the switcher or within the cloud            service        -   badges, which are bragging rights aspects    -   add audio muting or switch over for commercials and commercial        detection, or other events. add other audio source switchover as        a “secondary audio input during commercials. such as music from        other inputs, for example streaming form the internet, from        other HDMI inputs, Airplay, uPnP, other wireless sources (BT,        WiFi), other inputs—optical, RCA, audio jack, etc.    -   stream audio discussion from Pandora, Spotify, etc.    -   control the inputs via signaling such as pause while the primary        show is on.    -   use a simple input to invoke the switch, from a smartphone, from        “clapping” or other audio input, or other “touch” input on the        device itself (or a button)    -   stream content related to the TV content (or other content)        being watched.    -   performing video or audio analysis based upon past statistics        (audio level change for example) to determine the commercial        time periods (or other time periods of interest).    -   based upon input designating a commercial or a show period,        analyzing collected information to refine the commercials or        “‘secondary’ time period” period selection.        for example, storing a running average audio level, and upon        input “this is a commercial” determining the interval over which        the period is consistent based upon the audio or other ongoing        information (images, blackouts, etc.), then categorizing        statistics related to the tie periods and for time periods not        designated as commercial (or designated as not commercial) ad        setting up, or refining the rests.    -   general audio “leveling” i.e. no excessive loud or soft sounds        using a audio AGC approach, aid in the approach for folks which        some hearing loss, and amplify the audio, but dynamical adjust        it, so as to prevent excessive volume deviations (prevent the        wife from being woken up, no excessively loud explosions).    -   identifying specific types of sounds such as those not suitable        for children—cure words/profanity/explicit, and lowering volume        during those short periods.        (allowing parents to watch a movie with sleeping kids).    -   blanking video during inappropriate scenes having been detected,        or switching to another source, or modify the image so as to        “cover” inappropriate images (automatic censoring),    -   content filtering prevention . . . physically lock the device to        the TVs HDMI input in one embodiment, detect the rating or other        aspects of the content and prevent it from being displayed    -   display a message, and/or    -   shane to other content    -   blank screen and display a message    -   potentially with Picture picture.    -   wifi “security” guardian . . .    -   using BT to connect a device to the box, or a WiFi AP signal    -   request connection to the WiFi network    -   using the app on the device to configure WiFi for the visiting        device (enterprise or regular security) with guest access to        local resources, or other levels of access.    -   the device passing credentials to the visiting device via BT, or        via WiFi direct, or beacons.

1.4. Remote Extender

-   -   1.4.1. interfaces with the SH using Bluetooth for example    -   1.4.2. is detected by (or visa versa) BT beacons, in one        example, by SH to be associated with it, and or the SH        Application on the SHAD (smart hub application device)    -   1.4.3. receives the infrared (IR) commands via BT and generates        the IR commands to control the target device        -   1.4.4. can receive IR command via IR, and forward them, but            BTLe or standard BT may be better for indirect            communications and then IR for control.

What is claimed is:
 1. A wireless speaker device for executing commandsbased on voice recognition and detecting the presence of other wirelessspeaker devices, the wireless speaker device comprising: a Wi-Fi radio;a Bluetooth radio; one or more microphones; at least one loudspeaker; asignal generator for providing one or more audio information signals; anaudio information interface for providing the one or more audioinformation signals for use in the generation of respective acousticsignals to respective loudspeakers; and one or more processorsconfigured to execute program code to: detect the presence of a secondwireless speaker device, utilizing the Bluetooth radio; communicate withthe second wireless speaker device utilizing the Wi-Fi radio and/or theBluetooth radio, to provide simultaneous audio playback of streamingcontent utilizing both the at least one loudspeaker located within thewireless speaker device and at least one loudspeaker local to the secondwireless speaker device; communicate with one or more remote serversusing the Wi-Fi radio to receive streaming content for playback from aplurality of content sources; receive acoustic signals using the one ormore microphones; perform voice command recognition on the acousticsignals, to determine a connected speaker command; and perform one of aplurality of functions in response to the at least one connected speakercommand, wherein the plurality of functions includes selectingparticular content from one of the plurality of content sources as thestreaming content for playback to both of the wireless speaker devicesat the same time.
 2. The device of claim 1 wherein the acoustic signalsinclude the streaming content.
 3. The device of claim 1 wherein theacoustic signals comprise sounding signals that are inaudible signalsembedded in audio signals broadcast by at least one of the respectivespeaker devices, and the processor is further configured to determineproximity between the two wireless speaker devices.
 4. The device ofclaim 3 wherein the acoustic signals further include the streamingcontent.
 5. The device of claim 1 wherein the processor is furtherconfigured to execute program code to: modify the streaming contentplayed to both the at least one loudspeaker located within the wirelessspeaker device and at least one loudspeaker local to the second wirelessspeaker device, based upon a detected proximity of the second wirelessspeaker device.
 6. The device of claim 5 wherein the modification of thestreaming content includes reduction of a time delay between the outputsof the at least one loudspeaker located within the wireless speakerdevice and at least one loudspeaker local to the second wireless speakerdevice.
 7. The device of claim 5 wherein the modification of thestreaming content is based upon detected acoustic channel propertiesbetween the wireless speaker device and the second wireless speakerdevice.
 8. The device of claim 1 wherein the processor is furtherconfigured to: determine that the second wireless speaker device isproximal to the wireless speaker device, based upon both (a) detectionof at least one Bluetooth beacon and (b) detection of unique soundingsignals.
 9. The device of claim 8 wherein a determination that thesecond wireless speaker device is proximal further comprises estimationof a time delay for propagation of two or more acoustic signals.
 10. Thedevice of claim 1 wherein the processor is further configured to executeprogram code to: initiate a streaming content service selected from aplurality of streaming content services to which a user of the connectedspeaker is subscribed.
 11. The device of claim 10 wherein accesscredentials for at least one streaming content service are stored on aremote server.
 12. The device of claim 10 wherein access credentials forat least one streaming service are stored on the device.
 13. The deviceof claim 1, wherein the voice command recognition is performed using aremote server.
 14. The device of claim 13 wherein the remote server is athird party server providing a voice command recognition service. 15.The device of claim 1 wherein the processor further executes programcode to: use the one or more microphones and at least one loudspeaker toperform one of (a) a Voice over IP communication session or (b) a videocommunication session or (c) an IP Multimedia Core Network Subsystem(IMS) session, the session established with a remote device.
 16. Thedevice of claim 1 wherein the processor further executes program codeto: perform a network security function including at least approval ofaccess for a new device on to a WiFi network.